DOCUMENT RESUME 



ED 216 684 



IR 010 162 



AUTHOR 
TITLE 

INSTITUTION 

SPONS AGENC7 

REPORT NO 
PUB DATE 
CONTRACT 
NOTE 



EORS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Schulz, Russel E.j And Others 

The Development of Programming Design Guides. Final 
Report. 

Human Resources Research Organization, Alexandria, 
Va. 

Army Research Inst, for the Behavioral and Social 
Sciences, Alexandria, Va. 
ARI-RPr-80-24a 
Dec 79 

DAHCl : 9-78-C,-0010 

28p.;i For r/elated documents, see IR 010 163-164, ED 

203 864-868, and ED 205 202. 

J 

MF01/PC02 Plus Postage. 

♦Computer Oriented Programs; Flow Charts; Formative 
Evaluation; *Guides; * Instructional' Development; 
Military Training; *Models; Programing Languages; 
♦Systems Approach ■ ' 

Authoring Aids; * Instructional Systems Development 
Model 



ABSTRACT. 

The purpose of this research effort was to develop 
and evaluate a prototype system-independent Programming Design Guide 
(PDG) .for one of the 13 offline job ajids previously developed for use 
with the Instructional Systems Development (ISD) model. This PDG is 
intended to provide all of the guidance necessary for a user to 
implement the job aid on any of a large number of computer systems. 
The research effort. was divided into three major tasks: (1) the 
\ establishment of t'h'e content and format for Programming Design 
Guides; (2) the development of a guide; and (3) the evaluation of the 
guide. Advice is given for developing Programming Design Guides for 
dther blocks of the ISD model. Ten references are listed. (LLS) 
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FOREWORD 



The Computer-Based -Instructional- Systems Team of the US Army Research v 
Institute for the Behavioral and Social Scien</es (ARI) performs research and 
development in the area of educational techno! ogy that applies to military 
training* Of interest are methods for training individuals to develop and 
utilize instructional courseware in reasonable time, at acceptable .cost. 

This^Research, Product is one o f a^serie f which have b^en designed to 

support the implementation of the Instructional Systems Development Model (ISD, 
TRADOC Pamphlet 350-30). The -ISD Model is d step-by-step procedure for the 
analysis, design, development, implementation, and control of military course 
materials. A previous effort produced manual. Job Aids which are paper and 
pencil documents designed to provide "how to\do it" guidance for the ISD Model. 
This document is t part of a series of three developed to support the delivery 
of the manual Job Aids by computer. To accom&ish this research, ARI's resources 
were augmented by contract DAHC19-78-C-0010 witb the Human Resources Research 
Organization (UumRRO) « 

The contributions of personnel from ARI's Mfenpover and Educational Systems 
Technical t Area as well as those of Mr. Charles B. Marshall and Mr. Joseph P. 
-SSV5Ttr^*34§£ arch Facilities Support Group are acknowledged. Mr. Antonio J. 
Alameda, HuimRRtL also contributed to this resea«h effort. 
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THE DEVELOPMENT OF PROGRAMMING DESIGN GUIDES 



BRIEF 



Requirement: ~ ^ 

The purpose was to develop a language to translate an existing paper^nd 
pencil Job Aid onto a computerized delivery system* The Job Aid is one of a 
series developed previously to support users of the Instructional Systems 
Development Model (ISD) . 



Procedure: 

A Programming Design Language (PDL) was^xeeteci to describe the computer 
functions (e.g., computer/user intera^tioifs7 storage /retrieval of data, program 
branching, program management^airdcdlculations) required by the Job Aids (ARI 
Research Products 80^13^tKrough 80-18) . The PDL was designed to communicate 
to the cony^uter^pfogrammer in a language independent fashion so that the on-line 
ymput^r^ version of the Job Aid- can be delivered by any computer. 




Utilization: 

The Programming Design Guide may be used by computer programmers who are 
tasked ^ith programming the manual Job Aids. 
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BACKGROUND 

\ • C_ ^ 

/ The systems approach to training defines a process which focuses on the job that 
is ultimately to be perfo^ed and upon the individual who is to learn to perform that 
job. /The systems approach is just what the name, implies: a systematic ■ P™ess for 
specifying the desired products of training and selecting what wil be taught how it 
wiU be taught, what the presentation mechanism wiU be, and evaluating the effects of 
each phaseof the process. It focuses on student performance as a determinant of content. 
Its proper, application can hardly fail to improve" instruction where only incidental atten- 
tion has been given to these functions. However, after more than, two decades of research 
and development in this area, the full benefits of applying the systems apprdach to 
instruction have yet to be realized. , 

One attempt to standardize a definitive technology in this area-is the Instructional 
Systems Development Model (ISD 1). The ISD contains standardized rationale, termi- 
nology, and basic concepts of instructional^ systems. 

One of the primary difficulties of the ISD model and its manuals is that it empha- 
. sizes "what to do," not "how to do it." ISD manuals are intended to have general 
applicability. However, it has been clearly recognized by many experts that the same 
methodologies cannot be applied to the universe of training problems. How to do it 
may vary widely even though the specific processes or "what to do" may remain fairly 
SSacrosstoaining problems. Thus, much specific ^^^CtlZ^ 
are needed by training designers, developers, and managers, for ISD to be appropriately ^ 
implemented within the military training establishment. 

In this report, "how to do it" guidance/tools/procedures will.be referred to as author 
aids or job aids. An author is considered to be any member of an instructional staff 
charged with meeting the objectives of one or more components of the instructional 
development process. 

- A wide variety of aids exist, but the majority 'are more of the "what to do" than 
the "how to do it" .variety. Rather than providing actual help in performing the authoring 

• work, or even detailed how-to-do-it guidance, most of the existing author aids may serve 
simply to reinforce or broaden the guidance provided in ISD manuals. Care should be 
exercLd in the selection of aids to be integrated with ISD so that authors are not con- 
fronted with a confusion of different yet similar models, sets of jargon, procedures or 
forms. Existing general manuals differ from one another in that they may: 

• Include different steps or different names for steps. 

4 • Include different methods of accomplishing each step. 

• Provide different levels of specificity in the details included under each step. 

• Provide different formats for reporting the work accomplished under each step. 
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Thfis, to provide the most efficient aids to authors, the guidance found in some of these 
.manuals and guidebooks needs to be translated and integrated into the ISD framework, 
rather than referred to in its source form, 

Two author aids (2) developed in a previous project demonstrated Xhe utility of our 
approach toward assisting authors implement the ISD process. The major characteristics 
of these aids which account for their usefulness are the following: 

• The aids integrated existing ISD guidance. 

• The aids were query.-based and interactive. 

• They were based upon a realistic authoring needs assessment at an 
operational site. 

• Aids were developed and evaluated according to ISD guidance and 
recommended procedures. 

• The aids concentrated oh assisting authors dealing with structural and admin- 
istrative details permitting them to concentrate their efforts on the subject 
matter content of the materials. 

% . Current aids (3 through 8) cover the activities required for the first three phases 
of the ISD process. The 13 aids cover each block within these phases, excluding ISD 
Block LI, Analyze Job. Each job aid is composed of two documents— a descriptive 
authoring flowchart and a job aid manual. 

Descriptive Authoring Flowchart. The descriptive authoring flowchart is the primary 
document used in each job aid. It directs the user to specific guidance, examples and 
references provided in the job aid manual. 

Job Aid Manual. The Job Aid Manual provides the specific guidance, examples and 
references necessary to produce the product for the instructional systems development 
activity it covers. In addition, each Job Aid Manual contains one or more worksheets 
to use in the development of the product. 

- At this stage, it is important to determine if and with what effort these off-line job 
aids can be converted into analogous on-line versions, regardless of the CBI system to be 
used. What guidance is needed by computer programmers within the Army to prepare 
programs that can implement these aids on their systems? These questions form the basis 
of the research effort described in this report/ 
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PURPOSE 1 

* 

"^An earlier project (2) demonstrated, the utility of on-line query-based author aids , 
in the implementation of ISD Block II.2 (Develop Tests) and Block III.4 (Develop Instruc- . 
tion) The results of <Qiis 'study indicated the technical and practical merit of furthering 
the on-line, interactive aid concept to all applicable ISD Blocks and of prodding these - , 
aids in a relatively system independent form. This has led to the research effort descnbed 
in this report, the purpose of which Is: " 

i 

i . " 

■ to develop and evaluate a prototype system-independent 
•Programming Design Guide (PDG) for one of the off-line 
author aids- previously developed. This Programming Design 
Guide is intended to provide all of the guidance necessary for 
t a user to implement the author aid on any of a large, number 

of computer systems. • ^ 
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RESEARCH ACTIVITIES' 

* . / 

The research effort was divided into three major tasks: 
J 

TASK 1: Establish content and format for Programming Design Guides 
TASK 2: Develop a Programming Design Guide 
TASK 3: Evaluation of a Programming Design Guide 

Task 1: Establish Content ar\d Format for Programming Design Guides 

r • 

Task 1 activities were conducted, in see steps which sure described below. 



Step L Study Existing ProgramminfrDesign Languages 

'Programming Design Guides (PDG) are written in a Programming Design Language 
(PDL). Documents relevant to the development of programming design languages were 
examined to determine their usefulness for the PDL to be used in the research effotf. 
% None of these documents provided the needed guidance and so a suitable PDL had to 
be developed. 

Step 2. Analyze Job Aid Flowcharts for Interactive Processes 

1 In Step 2 the Descriptive Authoring Flowchart documents for the 13 author aids 
were carefully examined to identify all interactions between thS user-and the aids. The 
purpose of this examinaticm was to provide greater assurance that the Programming. 
Design Language would encompass the interactions required by- any individual aid. 

■v 

Step 3. Analyze Narrative Description of Job Aids for Additional Programming Design 
Language Requirement 

- The narrative description provided in the job aid mtjnuals for each of the 13 author 
aids was examined to identify any additional programming design language requirements 
not obvious in the flowcharts. No additional programming design language requirements 
were identifies as a result of this examination. 5 

-Step 4. Define/Describe Computer Functions Required for Job Aids 

Computer functions such as computer/user interactions, storage/retrieval of data, 
program branching, program management, and calculations required for the aids were 
defined and described to the developers of the Programming Design Language. The 
identification in Steps 2 and 3 of the interactive processes within aids provided input * 
to the definition of the computer functions. S - 
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Step 5. Develop Programming Design Language to Satisfy f»nc»o««« ^u.,.,..— _ 

The basic requirement of the Programming Design Language is that it 1 be to 
natural language so as to facilitate communication between defers and the computer pro- 
^mers of L author aids independent of the computer system and Pjo^ng 
Lguage used. The Programming Design Language Is noj a ^programming language .It is 
LDseuLcomputer language which is used to describe the design specification for certain 
claTs of inSSiv computer programs. It was assumed that programmers using the 
Primming Design Guides would be skilled in an appropriate programing language. 
AfteTompTetion S programming the resulting on-line version of the pb «djjto b. 
used by military instructional development personnel to perlorm the ISD activ ties 
1 Quired in thVaid. PDL is designed to communicate to the programmer in a language 
independent fashion. \ 

\ 

■Step 6. Determine Format for the Programming Design Guide 

The activities in the previous steps of Task 1 enabled us to determine the optimal 
format for the Programming Design Guide. The format developed proves for display 
of specific block(s) of the flowchart, the labels, commands, tags and comments nee- 
essary for the programming of the author aid. An example of the format that was 
developed and used is shown in Figure 1. 



Figure 1 



Facing Pages of Section VI of the Programming Design Guide 
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BtOCX 



9t 



9b 



9c 



00MMAMO 



saow 


What iCS art you working vlch? 


jaccxjt 


JWSCODE.IO 


saow 


What is cht skill level? 


Acczrr 


SKILL, 1,4 


SBOV 


You are working vlch M)S /SMOSCODE/ 
md chi skill livtl Is /SKILL/. 
Is this comic? 


OtCXDI 


bLock9i, blockBa; blockos 


SBOV . 


(ctxc9i) 



WAIT 



ITEBATZ 
SHOW 



!ftXt 

SBOV 

, T XCU>Z 

SBOV 

WAIT 



DP , I , ZkJXJXrttJOS I TIONS, 
Ducy potltlon /DP/ Is /SDUTTCODE(DP)/ 



OP 

Are chase ducy positions correct? 
blocklOa, block9c 



COMMENTS 



***roll on all ducy potltlon 
designators onto cereinal. 
The end product should be a 
Use of ducv potltlon 
designators. 



(caxt9c> 
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(cexc9s) 



( cexc9c) 



In previous LSD Blocks 1c was established that chtrt m 
/NUMBE10P_DUTrjFOSinONS/ duty positions. 

f?*ESS NEXT# to 9« i llsc of che ducy positions chat vera 
recorded In earlier ISD Blocks. I 



Ic Is excre**ly issportaot that before you nake any additions, 
deletions, or changes In ducy poeldons chVc you check vlchi 

- Tour eupsrvisor 

- The Individual (s) who prepared che Critical Taek Llsc 
(ISD 1,2) 

- The individual (s) who performed taek analyele (ISD 1.3). 
If chey agree vlch your suggesced additions, deleclone, o 
changes you wlU be allowed co enter then Inco che termln. 
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Task 2: Develop the Programming Design Guide 

The activities in Task 2 were conducted in five steps as described 



Step r. Select Aid for Programming Design Guide Development 

In this step a specific job aid was selectedfor which the Programming Design Guide 
was to be developed The ob aid for ISD Block 1.5 Select Instructional Setting was selected 
because of it is representative of the other aids, in terms of function requirements com- 
PlexTy and dSS in Programming Design Guide development. Approval of this selection 
was Sen obtained from the Army Research Institute's contracting officer's representative. 

Ste p 2. Translate Job Aid Into Programming Format 

Step 2 involved the three activities described below. 

• 'Revision of the Flowchart for the job aid for TSn Block 1.5 Select Instructional 
Setting. The flowcharts as currently presented in the author aids are at a Jevel [of det ail 
Efficient to communicate off-line with military instructional development personnel. They 
equSe th inXctional developer to provide and record information that can automaUc^ly 
bXaae available in computer versions of the author aids. Therefore one of the activities 
to Step 2 was to revise the flowchart of the ISD 1.5 aid for its intended use as . part of the 
PWaLming Design Guide. In the revision process, care was taken to insure that the basic 
intent of the original flowchart was not affected. 

• \ Dat a Storage/Retrieval. In this activity we identified 'the type of date that would 
normally be avaUabffto individua ls charged with the ISD activity of selec ing the mstouc- 
ST' setting for each critical task, the type of data that would be generated as a result 
of tee plocls of Sleeting the instructional setting, and an identification of where this 
col ecte P d data would be required in conducting the ISD. activities of later ISD ^Btock* We 
also identified when the data should be made available to the user and the format for its 
presented For example, in selecting instructional settings for critical mditery tasks .* 
rneceSaTto have a list of these critical tasks and the duty positions within the MOS in 
Ih^S are performed. The critical task listing and the duty positions comprising the 
MOS^arrdeSLined in'earlier ISD blocks. If these earlier blocks had been incorporated 
toto online vSions that information would already be to the computer. The Programming 
Sesign Guide would then need only to provide guidance in methods of accessing thi _ infor- 
mation However, since the information was not already resident in the computer, the 
Camming Design Guide provides guidance for placing the necessary information from 
2lS?SD blocks toto the computer. The Programming Design Guide , £ ^£>™on 
for permanentstorage of data, such as the instructional setting selected for each task, 
which would be used in later ISD blocks. 

. Id entification and D evelo pment of Basic Displays . Basic displays include materials 
such as a discussion of the puipose of a particular activity techniques for P^™ 1 ^ that 
the activity, sources of information, and questions asked the user. .The display format that 
L uid Snds upon the specific display to be presented and its purpose. In addition, a 
dispTy mav require an actiVe response from the user (answering a question). Because of 
th £T' limitations of the typical computer display devices, a decision was made that 
no di play identified in the ProgLnmtog Design Guide would be greater than 10 lines with 
J nStaL 40 alphanumeric characters per line. In addition, it was decided that since 
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many computer display devices do not have graphic capabilities, that all tables in the 

printed (off-line) version of the author aid would be presented in narrative form in the 

Programming Design Guide displays. All basic displays were thus identified in this step, 

and the format for their presentation to the user of the on-line author aid was established. * 

-> 

Step 3. Create frogramming Specifications in Program Design Language 

The expanded flowchart, data storage/retrieval requirements, and basic displays iden- c 
tified or developed in Step 2 were translated into programming specifications in terms 
of foe Programming Design Language. These were prepared and incorporated into the 
Programming Design Guide. 

Step 4. Prepare Draft Programming Design Guide 

The Programming Design Guide (9) is organized into six sections as described below. 
Introduction 

Programming Design Language. This section describes the commands 
used in the Guide and provides guidance and examples of how each 
is used. Figures 2 and 3 show two pages of Section II of the Pro- 
gramming Design Guide. These pages describe the "SHOW" and 
the "ITERATE" commands. 

Programming Flowchart . The flowchart included in Section HI is 
an overview of the programming requirements for the entire program. 
Figure 4 shows a sample section of the flowchart contained in Sec- 
tion II of the Programming Design Guide. 

Variables Used in the Programming Design Guide. This section 
provides an alphabetical listing of all of the variables used in the 
program for implementation of the job aid for Selecting Instruc- 
tional Setting. Figure 5 shows the first page of Section IV. 

Setup Material. In this section guidance is provided for setting 
some variables to predetermined values and establishing various 
strings and arrays. The first page of Section V of the Program- 
ming Design Guide is shown in Figure 6. 

Programming Specifications. Section VI is the heart of the Program- 
ming Design Guide. As previously stated, it contains specific block(s) 
of the flowchart, the labels, commands, tags, and comments necessary 
for the programming of the Select Instructional Setting job aid. 
Figure 1 shows two facing pages of Section VI. 

Step 5. Prepare Supplemental Guide 

In addition to the Programming Design Guide, a hard copy supplemental guide was 
prepared, This supplement (10) contains textual material from the job aid that permits 
users of the on-line aid to review previously seen material. 



SECTION I: 
SECTION II: 

SECTION III: 
SECTION IV: 
SECTION V: 
SECTION VI: 
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Page From Section 1 1 of the Programming Design Guide 
Describing the "SHOW" Command 



Implementation of Specific Text 

Since computer systems differ among installations, the PDL includes the 
ability to specify where a PDL phrase should be consistently replaced 
with a phrase appropriate for a specific computer system. The PDL phrase 
is enclosed in a pair of //'signs. 

Example 

SHOW //PRESS NEXT// TO CONTINUE \ 

#PRESS NEXT// means that the user signals readiness to proceed 
by pressing a function key or typing a command. 

In one implementation, this message might be "PRESS CARRIAGE RETURN 
to continue. 11 

For another system, the message might say: f, HIT CARRIAGE RETURN 
to continue. 11 - \ 
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Figure 3 

Page From Section II of the Programming Design Guide 
Describing the "ITERATE" Command 



ITERATE index, first value, last value [, increment] 



The ITERATE command specifies the beginning of an iterative loop structure. 
The variable "index" is first set to the value "first value. Subsequent 
commands are processed in the normal manner, until a NEXT index command 
is encountered. The loop is then 'restarted with the variable >dex first 
having the value of "increment" added to it. The increment value defaults 
to one if unspecified. The loop terminates when the value of "index" becomes 
greater than "last value." Control then transfers to the next command 
after "NEXT index." 



Examples 

•ITERATE I', 1 , 10 



NEXT I 

ITERATE J , 1 , 9 , 2 



****J will be 1, 3, 5, 7 and 9 
during five loop passes. 



NEXT J 
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Figure 4 

Sample Section of Flowchart Contained in 
Section III of Programming Design Guide 



22 



Ask Question 4. 
Record "Y" or "N" 
as Appropriate. 
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Assign-Task Initially to 
INSTITUTIONAL . 
Instructional Setting. 




Examine Next Task 
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Figure 5 

Example of Variables Used in the 
Programming Design Guide 



Section IV 

VARIABLES USED-IN THE PROGRAMMING DESIGN GUIDE 



This section of the Programming Design Guide provides an alphabetical 
listing of variables used in the program. Generally, the variable names are 
self-explanatory. However, where needed, an explanation of what the variable 
is used for is provided.. 

Keep in mind that these variable names are used only to communicate with 
programmers who are using the Programming Design Guide. Feel free to rename 
any variable. , * * * 



DP 

'DPNEW 
$DUTYCODE 
$DUTYC0DE2 

DU TY CO DE_MA XIMUM 

$ ESTIMATE 



FIRST TIME 



FSETTING 



HP CRITERION 



"$INPUTLINE,100 




A numeric variable used to index the duty position 
array* 

A numeric variable used to count the number of duty 
positions during the modification process. 

An array of alphanumeric variables used to define the 
duty positions* 

A temporary array of alphanumeric variables used in 
the program when the user is modifying the duty 
position designations* 

A numeric variable which defines the maximum number 
of alphanumeric characters that can make up a duty 
position title. 

Temporary variable that stores yes/no indicating 
whether percentage performing task is an estimate 
or not* 

Variable used to determine if the first unassigned 
task is being re-evaluated. 

An array of numeric variables used to store the final 
designations of instructional settings. The numbers 
stored will be a 1, 2, or 3. 



Criterion value for percentage of spldiers who must 
erform a task 'before the task is identified as a 
;h performance task. 11 

An alphanumeric variable that is to store temporarily 
the userV. input duty code designation. It is 
limited to 100 characters. 



Figure 6 

Example of "Setup Material" Contained in 
Section V of the Programming Design Guide 



Section V 

"setup material 



To facilitate the programming of the computer versioa of the Job Aid 
for Selecting Instructional Settings, it is necessary that you first program 
(in your programming language) setup material. A guide for the necessary 
setup material is shown below. You must, of course, establish your own value 
for MAXIMUM TASKS, MAXIMUMJDUTY_POSITIONS, and NUMBERJ)F^DUTY_POSITIONS. In 
addition, you will provide your ^own task ID numbers and task titles for 
$TASKC0DE(1) thru $TASKCODE(H) ~ limit 40 characters — and your duty positions 
for $DUTYC0DE(1) thtu DUTYCODE(n) — limit 30 characters. 



MAXIMUM_TASKS = 24 
MAXIMUM_DUTY_P0SITI0NS = 15 
TA S KC0DE_MAXEMUM =40 
DUTYCODEJMAXIMUM * 30 

$ TAS KCO DE ( MAXIMUMJIAS KS ) , TASKCODE_MAXIMUM 
$DUTYC0DE(MAXIMUM_DUTYJP0SITI0NS) , DUTYCODE_MAXIMUM 
$DUTYC0DE2 (MAXIMUM_DlOT_P0SITI0NS , DUTY C0DE_MAXIMUM) 
$MDSC0DE,10 

I S R_DUTY (MAXIMUM_DUTY_P0SITI0NS , MAXI MUM_TAS KS ) 
« ISR%(MAXIMUM_TASKS) 
I5R%_E(MAXIMnM TASKS) 
S ETTING ( MAXI11UMJTASKS ) 
FSETTINGS (MAXI&UM_TASKS) . 
ISR_QUESTI0N(14, MAXIMUM_TASKS) 
$INSTR_SETTING(3) ,20 
$INSTR_SETTING(1) « "INSTITUTIONAL" 
$INSTR_SETTING(2) = "S 0 J T tf 
$INSTR_SETTING(3) = "SELF STUDY" 
NUMBER OF TASKS = 24 



SET 
SET 
SET 
SET 

$STRING 
v $ STRING 
^STRING 
$ STRING 
ARRAY 
ARRAY 
ARRAY 
ARRAY 
ARRAY 
ARRAY 
$STRING 
$SET 
$SET 
$SET 
SET 

$SET $TASKC0DE(1) = 574-2058 



$SET $TASKC0DE(2) = 587-0025 

$SET $TASKC0DE(3) - 587-0032 



$SET $TASKC0DE(n) = 587-1027 



Operate Rddio Test Set AN/VRM-1 to 
Test Modules in AN/VRC-12 Series 
Radio Sets 

Repair Radio Set, AN/PRC-25/77 

Systems Troubleshooting Radio Set, 
AN/VRC-12 including C-2742/VRC to a 
Defective Component, Cable or 
Accessory 

Verify installation of. Radio Set 
AN/VRC-46 in a Tracked Vehicle 
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Task*3: Evaluation of the Programming Design Guide 

In Task 3 we formatively evaluated and revised the Programming Design Guide 
imJSaSt Task 2 Te purpose of this Task was to obtain whatever information we 



Guide, 



The evaluationwas conducted to three steps. The first two steps mvofced 
installation of the Job Aid for Selecting Instructional Setting ^^^^o 
Based Instruction (CBI) systems using the Programming Des^ iGjk, In the third step 
the Programming Design Guide was indirectly evaluated by us mg .the p 
gram to simulate selection of instructional settings for a sample of military tasks. A aescrip 
tion of each evaluation step and its outcome follows. 

Step 1. Formative Evaluation 

• Procedure As major portions of the programming specifications in the draft Pro- 
erarnmiSD^Guid Twere completed, a project staff member used the Guide to program 

narrative displays were not produced in their entirety. 

. p PS „lt. s of the Evaluation. The programmer had no difficulty in .using the 
Progra mming Design Guide. The programmer was thoroughly familiar with the author 
S?^d bv tKrogramming Design Guide but had not been directly involved- in the 

ately corrected: It required approximately 100 hours to program the aid in BASIC using 
the Programming Design Guide. - 

Step 2. Formative Evaluation by U.S . Army Research Institute Systems Analyst 

• Procedure After each portion of the programming specifications (Section VI, 
Proeramln^ign Guide) had been programmed into the micro-computer and revisions 
m a T?tw£ sent to the U S Army Research Institute. A systems analyst was provided 
tots'ta^Uhe aW using the Programming Design Guide.' This systems analyst was m no 
^^^JSti^^t of either the Programming Design Language or the 
P?o~?n7Di gn Guide. The computer system used for installing the aid was a\ 
Programming resign UL " UC - ( ppT The CRT has a 50 x 20 matrix screen 

CDC 3300 computer and a CDC 210/11 CRl. ine Oiti na. a ot 
which is limited to 1000 characters. It produces only upper case letters in white on a 
tock screeT In this evaluation the Programming Design Language was translated into 
FORTRAN. 



1 The ARI systems analyst was Charles F. Marshall. 
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The specific procedure used in this step of the evaluation was as follows: 



— The systems analyst programmed the job aid into the computer using only the 
Programming Design G uide (without further assistance). 



— An anecdotal record was maintained. „ 

— The tim'3 taken to program the aid was recorded. 



— Telephonic communication was maintained between HumRRO and the systems 
analyst on almost a daily basis to discuss progress. > 

• Results of the Evaluation. No major problems arose in the installation of the job 
aid on the computer using the Programming Design Guide. The systems analyst did not 
always agree with the wording in some of the narrative displays shown in the Programming 
Design Guide. In a few* cases he asked to change the wording. This was permitted unless 
At changed the intent of the narration*. However, no programming problems occurred that 
required a major revision of the Programming Design Guide. 

After each portion of the aid had been installed into the computer the systems 
analyst and the project director reviewed the resulting program on-line to identify any „ 
additional problems resulting from an incorrect interpretation of the Programming Design 
Guide. No major problems w£re discovered as a result of this review. Approximately 
115 hours were required for the systems analyst to install the job, aid on the CDC 3300 
computer. This time involved the following activities: 

— Translating Programming Design Language into FORTRAN 36 hours- 

— Keypunching and assembling decks 18 hours 

— Debugging and revisions 61 hours 

* TOTAL 115 hours 



Step 3. Formative Evaluation of On-line Aid 

• Procedure . After the entire Job Aid for SelectingJnatructional.Setting had'be*en 
installed into-the- CDG^300-corai^t^;^Ha"fiadrbIeen "debugged Aa-finaKevaluation of 
the on-line program was conducted. This evaluation was performed by the alternate ARI 
contracting officer's representative. 1 This individual used the on-line program in much 
the same way as would an intended military user; that is, a simulated task to select instruc- 
tional settings was performed by a COR. The purpose of this part of the evaluation was 
to identify any additional "bugs" in either logic or narrative material. 

• Results of Evaluation . Only minor changes in wording of narrative material were * 
required as a result of this evaluation effort. The evaluation required approximately five 
hours to complete. 



The alternate ARI COR was Dr. Melissa Berkowitz. 

m 
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CONCLUSIONS AND RECOMMENDATIONS 



The following are- conclusions and recommendations glewied from the formative 
evaluation of the Programming Design Guide, 

• The Programming Design Guide for the Select Instructional Setting job aid was 
an easily used and effective tool for installing this aid into two widely different computer 
systems. Neither the project programmer, nor the systems analyst encountered any. dif- 
ficulty in translating the Programming Design Guide into a meaningful and effective on- 
-line program. The timetequired for installing tlie job aid on the computer using the 
.Programming Design Guide was 100 hours for the project programmer. It required 115 
hours for the systems analyst tolnstall the aid on the computer. In neither case is the 
^time'for installation of the aid on 'the computer excessive. They are probably representa- 
tive of the time that would be required for installation on other computer systems. 

• The logic contained in the on-line programming of the job aid is cotaplex. A 
misinterpretation of the guidance contained in the Programming Design Guide can result 

in an error in the on-line program which might not be readily apparent to the programmer. 
For this reason it is strongly suggested that detailed guidance is needed for checking the 
accuracy of the programmed, on-line job aids. This guidance should be in the form of 
leading the "debugger" through each step of the program, stating exactly what to enter 
and the result of each entry. In our opinion, a separate "debugging" guide is required 
for each Programming Design Guide. 

• We determined that the on-line version of the author tad produced by using the 
Programming Design Guide was an effective instructional systems, development tool for 
selecting instructional settings for training military tasks. There is no apparent reason why 
Programming Design Guides developed for other ISD blocks would not be equally effective. 
Therefore, it is recommended that Programming Design Guides be developed for the remain- 
ing blocks of the ISD model. 



• The guidance provided in the next section should be sufficient to develop Pro- 
gramming Design Guides for any specific ISD block. However, the effort required to develop 
any PDG is, considerable. Certainly it would not be cost-effective for every instructional 
systems development installation to develop their own Programming Design Guides for the 
ISD blocks. A master set should be developed covering all of the ISD blocks and these 
should be distributed to* the appropriate ISD installations for implementing author aids 
on their own computer systems, ■ 
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GUIDANCE FOR PROGRAMMING DESIGN GUIDE DEVELOPMENT 



- As stated in the preceding section of this report, we feel that the present research 
activity indicates the desireability of developing Programming Design Guides for other 
blocks of the ISD model. Further development ofcProgramming Design Guides will not. 
require all of the steps performed in the initial dj^opment activity. fci our opinion, 
only the following steps need to be perfoim^whe^dfeveioping Programming Design 
Guides for the remaining blocks. 

Step 1. Analyze Job Aid Flowcharts for Interactive Processes 

Step 1 involves the careful study of the author aid to obtain an overview of the ^ 
requirements of the Programming Design Guide to he developed. 

Step 2. Translate Job Aid Into Programming Format 

• R evision of Flowchart. The flowchart contained in the printed (off-line) version of 
the author aid should be studied and revised as necessary to provide sufficient assistance 
for input into a computer system. - .* 

• Data Storage/Retrieva l. In this activity "the data* that should be available to the user 
of the resulting on-line program should be identified and provision made within the Pro- 
erammine Design GuHe for making the data available. In addition, the type of data to be , 
generated within thv - -gram should be identified and specific points within the program 
•at which it should be made available should also be identified. Provisions for storage and 
retrieval of the data should be included in the Programming Design Guide. 

• THpn t.ifi«rtion-and Development of Basic Displays. Basic displays of materials such 
as a discussion of the purpose of a'particular activity, techniques for performing the ^yrty, 
sources of information, questions asked the user, and other information specific : to the author 

' aid for which the Programming Design Guide is bsing developed should be identified for 
use in step 3. 

St ep 3. Create Programming Specifications in the Prog ramming Design Unguage 

The revised flowchart, data storage/retrieval requirements, and basic displays identified 
or developed in Step 2 are translated into programming specifications m the Programming 
Design Language for inclusion in the Programming Design Guide. 



/ 
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Step 4. Prepare Draft Programming Design Guide. 

j> ■ 

Using the information identified or developed in Steps 1 through 3 a draft version 
^of the Program .ning Design Guide is prepared in^Step 4. The Programming Design Guide 
should be organized into six sections similar to those in the Guide developed in this 
research effort. The" sources of information for each section 'is as follows: * 

SECTION I: Introduction, Use the introduction in the original Guide. 

Programming Design Language. All Programming Design Language com 
mands included in the present Guide should be employed. , 

Programming Flowchart . Specific flowchart blocks would be derived 
from Step 2 above. However, the format shouldTbe the same as that 
in the present Guide. 

Variables Used in the Programming Design Guide . These variables 
would be developed in Stip 3*. — ' .f" ?/" * 

Setup -Material. _ Essentially, thjssame format as in the present Guide. 
Specific variables that muSt be set to predetermined values and the 
various strings and -arrays that need to be established would be 
-■ identified in Step 2. . 

SECTION- VI: Programming Specifications. This is to be obtained from activities 
in Step 3. . - f 

Each of the resulting Programming Design Guides should be ac ompanied by a guide 
for "debugging" the on-line program.* To provide a standardized format nomenclature, and 
methodology for implementing the various Programming!) esign Guides, it is strongly 
recommended that the complete set be prepared at a central facility. 



SECTION II: 



SECTION III: 



SECTION IV:' 



SECTION V:. 



* 
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